Skip to content

[Bots] Managed robots.txt update #23213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jun 24, 2025
Merged

[Bots] Managed robots.txt update #23213

merged 5 commits into from
Jun 24, 2025

Conversation

@patriciasantaana
Copy link
Contributor

@patriciasantaana patriciasantaana commented Jun 24, 2025

Summary

document prepending existing robots.txt on websites

Documentation checklist

@github-actions github-actions bot added the product:bots Related to Bots product label Jun 24, 2025
@hyperlint-ai
Copy link
Contributor

hyperlint-ai bot commented Jun 24, 2025

Howdy and thanks for contributing to our repo. The Cloudflare team reviews new, external PRs within two (2) weeks. If it's been two weeks or longer without any movement, please tag the PR Assignees in a comment.

We review internal PRs within 1 week. If it's something urgent or has been sitting without a comment, start a thread in the Developer Docs space internally.

PR Change Summary

Updated the managed robots.txt documentation to clarify how it interacts with existing robots.txt files.

  • Clarified the behavior when an existing robots.txt file is present.
  • Specified the response for websites without a robots.txt file.
  • Improved the overall structure for better readability.

Modified Files

  • src/content/docs/bots/additional-configurations/managed-robots-txt.mdx
How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2025

This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:

Pattern Owners
/src/content/docs/bots/ @patriciasantaana, @cloudflare/pcx-technical-writing
* @cloudflare/pcx-technical-writing

@patriciasantaana patriciasantaana marked this pull request as ready for review June 24, 2025 21:05
@patriciasantaana patriciasantaana requested a review from a team as a code owner June 24, 2025 21:05
@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2025 •
edited
Loading

Preview URL: https://1d630fe2.preview.developers.cloudflare.com
Preview Branch URL: https://patricia-pcx17991-robotstxt-changelog.preview.developers.cloudflare.com

Files with changes (up to 15)

Original Link Updated Link
https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ https://patricia-pcx17991-robotstxt-changelog.preview.developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/

deadlypants1973
deadlypants1973 approved these changes Jun 24, 2025
View reviewed changes
@patriciasantaana patriciasantaana enabled auto-merge (squash) June 24, 2025 23:09
@patriciasantaana patriciasantaana merged commit 41f4c8b into production Jun 24, 2025
11 checks passed
@patriciasantaana patriciasantaana deleted the patricia/pcx17991-robotstxt-changelog branch June 24, 2025 23:21
sdnts pushed a commit to sdnts/cloudflare-docs that referenced this pull request Jul 24, 2025
@patriciasantaana @deadlypants1973 @sdnts
[Bots] Managed robots.txt update (cloudflare#23213)
c76fa8e 
* text edits

* changelog

* changelog

* Update src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Co-authored-by: Kate Tungusova <[email protected]>

* Update src/content/docs/bots/additional-configurations/managed-robots-txt.mdx

Co-authored-by: Kate Tungusova <[email protected]>

---------

Co-authored-by: Kate Tungusova <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deadlypants1973 deadlypants1973 deadlypants1973 approved these changes

Assignees

No one assigned

Labels

product:bots Related to Bots product size/xs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@patriciasantaana @deadlypants1973 @pedrosousa @dcpena @haleycode